Training policy neural networks using path consistency learning

ABSTRACT

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for training a policy neural network used to select actions to be performed by a reinforcement learning agent interacting with an environment. In one aspect, a method includes obtaining path data defining a path through the environment traversed by the agent. A consistency error is determined for the path from a combined reward, first and last soft-max state values, and a path likelihood. A value update for the current values of the policy neural network parameters is determined from at least the consistency error. The value update is used to adjust the current values of the policy neural network parameters.

CROSS-REFERENCE TO RELATED APPLICATION

This application is a continuation of International Application No.PCT/US2018/019416, filed Feb. 23, 2018, which claims the benefit under35 U.S.C. 119 of Provisional Application No. 62/463,562, filed Feb. 24,2017, both of which are incorporated by reference.

BACKGROUND

This specification relates to reinforcement learning.

In a reinforcement learning system, an agent interacts with anenvironment by performing actions that are selected by the reinforcementlearning system in response to receiving observations that characterizethe current state of the environment.

Some reinforcement learning systems select the action to be performed bythe agent in response to receiving a given observation in accordancewith an output of a neural network.

Neural networks are machine learning models that employ one or morelayers of nonlinear units to predict an output for a received input.Some neural networks are deep neural networks that include one or morehidden layers in addition to an output layer. The output of each hiddenlayer is used as input to the next layer in the network, i.e., the nexthidden layer or the output layer. Each layer of the network generates anoutput from a received input in accordance with current values of arespective set of parameters.

SUMMARY

This specification generally describes a reinforcement learning systemthat trains a policy neural network that is used to select actions to beperformed by a reinforcement learning agent interacting with anenvironment.

According to a first aspect there is provided a method for training apolicy neural network used to select actions to be performed by areinforcement learning agent interacting with an environment byperforming actions from a pre-determined set of actions, the policyneural network having a plurality of policy network parameters and beingconfigured to process an input observation characterizing a currentstate of the environment in accordance with the policy networkparameters to generate a score distribution that includes a respectivescore for each action in the pre-determined set of actions, and themethod comprising: obtaining path data defining a path through theenvironment traversed by the agent, wherein the path is a sequence froma first observation to a last observation, and wherein the path dataincludes a plurality of observation-action-reward tuples, wherein theobservation in each tuple is an observation characterizing a state ofthe environment, the action in each tuple is an action performed by theagent in response to the observation in the tuple, and the reward ineach tuple is a numeric value received as a result of the agentperforming the action in the tuple; determining a combined reward fromthe rewards in the tuples in the path data; determining a first soft-maxstate value for the first observation in the path in accordance withcurrent values of state value network parameters of a state value neuralnetwork; determining a last soft-max state value for the lastobservation in the path in accordance with the current values of thestate value network parameters of the state value neural network;determining a path likelihood for the path in accordance with currentvalues of the policy parameters; determining a consistency error for thepath from the combined reward, the first and last soft-max state values,and the path likelihood; determining a gradient of the path likelihoodwith respect to the policy parameters; determining a value update forthe current values of the policy parameters from the consistency errorand the gradient; and using the value update to adjust the currentvalues of the policy parameters.

In some implementations, determining the combined reward comprisesdetermining a discounted sum of the rewards in the tuples in the path.

In some implementations, the state value neural network is based on a Qneural network, and determining the first soft-max state valuecomprises: processing the first observation using the Q neural networkin accordance with current values of Q network parameters to generate arespective Q value for each action in the predetermined set of actions;and determining the first soft-max state value V for the firstobservation that satisfies:

${V = {\tau\;\log{\sum_{a}\left( {\exp\frac{Q(a)}{\tau}} \right)}}},$where the sum is over the actions a in the predetermined set of actions,Q(a) is a Q value output of the Q neural network for the action, and τis a constant value.

In some implementations, the state value neural network is based on a Qneural network, and determining the last soft-max state value comprises:processing the last observation using the Q neural network in accordancewith current values of Q network parameters to generate a respective Qvalue for each action in the predetermined set of actions; anddetermining the last soft-max state value V for the last observationthat satisfies:

${V = {\tau\;\log{\sum_{a}\left( {\exp\frac{Q(a)}{\tau}} \right)}}},$where the sum is over the actions a in the predetermined set of actions,Q(a) is a Q value output of the Q neural network for the action, and τis a constant value.

In some implementations, the policy neural network is based on a Qneural network, and processing an observation using the policy neuralnetwork to generate a score distribution that includes a respectivescore for each action in the pre-determined set of actions comprises:processing the observation using the Q neural network in accordance withcurrent values of Q network parameters to generate a respective Q valuefor each action in the predetermined set of actions; determining asoft-max state value for the observation that satisfies:

$V = {\tau\;\log{\sum_{a}\left( {\exp\frac{Q(a)}{\tau}} \right)}}$where the sum is over the actions a in the predetermined set of actions,Q(a) is a Q value output of the Q neural network for the action, and τis a constant value; and determining a score distribution π for theobservation that satisfies:

${{\pi(a)} = {\exp\left\{ \frac{{Q(a)} - V}{\tau} \right\}}},$where a is an action in the predetermined set of actions, π(a) is ascore for the action in the score distribution, Q(a) is a Q value outputof the Q neural network for the action a, V is the soft-max state valuefor the observation, and τ is a constant value.

In some implementations, determining the path likelihood for the pathcomprises: processing each observation other than the last observationin the path using the policy network to determine a respective scoredistribution in accordance with the current values of the policy networkparameters; determining, for each observation other than the lastobservation in the path, a selected action score, wherein the selectedaction score is the action score for the action in the same tuple as theobservation in the score distribution for the observation; anddetermining a discounted sum of the logarithms of the selected actionscores.

In some implementations, the value update for the current values of thepolicy parameters is a product of the consistency error and thegradient.

In some implementations, the method further comprises: determining agradient of the first soft-max state value with respect to the Q networkparameters; determining a gradient of the last soft-max state value withrespect to the Q network parameters; and determining a value update forthe current values of the Q network parameters from the consistencyerror, the gradient of the first soft-max state value, and the gradientof the second soft-max state value for the last observation; and usingthe value update to adjust the current values of the Q networkparameters.

In some implementations, the method further comprises generating thepath on-policy by selecting the actions to be performed in response tothe observations in the path using the policy neural network and inaccordance with the current values of the policy network parameters.

In some implementations, the method further comprises sampling the pathdata defining the path from a replay memory storing data generated as aresult of interactions of the agent with the environment.

In some implementations, the method further comprises providing thetrained policy neural network for use in selecting actions to beperformed by the reinforcement learning agent interacting with theenvironment.

According to a first aspect there is provided a method for training apolicy neural network used to select actions to be performed by areinforcement learning agent interacting with an environment byperforming actions from a pre-determined set of actions, the policyneural network having a plurality of policy network parameters and beingconfigured to process an input observation characterizing a currentstate of the environment in accordance with the policy networkparameters to generate a score distribution that includes a respectivescore for each action in the pre-determined set of actions, and themethod comprising: obtaining path data defining a path through theenvironment traversed by the agent, wherein the path is a sequence froma first observation to a last observation, and wherein the path dataincludes a plurality of observation-action-reward tuples, wherein theobservation in each tuple is an observation characterizing a state ofthe environment, the action in each tuple is an action performed by theagent in response to the observation in the tuple, and the reward ineach tuple is a numeric value received as a result of the agentperforming the action in the tuple; determining a combined reward fromthe rewards in the tuples in the path data; determining a first soft-maxstate value for the first observation in the path in accordance withcurrent values of Q network parameters of a Q neural network;determining a last soft-max state value for the last observation in thepath in accordance with the current values of the Q network parametersof the Q neural network; determining a path likelihood for the path inaccordance with current values of the policy parameters; determining aconsistency error for the path from the combined reward, the first andlast soft-max state values, and the path likelihood; determining agradient of the path likelihood with respect to the policy parameters;determining a value update for the current values of the policyparameters from the consistency error and the gradient; and using thevalue update to adjust the current values of the policy parameters.

According to a third aspect, there is provided a system comprising oneor more computers and one or more storage devices storing instructionsthat when executed by the one or more computers cause the one or morecomputers to perform the operations of any of the previously describedmethods.

According to a fourth aspect, there are provided one or more computerstorage media storing instructions that when executed by one or morecomputers cause the one or more computers to perform the operations ofany of the previously described methods.

Particular embodiments of the subject matter described in thisspecification can be implemented so as to realize one or more of thefollowing advantages.

The reinforcement learning system as described in this specification canbe trained using both on-policy and off-policy path data in order toupdate, and preferably improve, an action selection policy based onwhich actions may be selected for performance by an agent whichinteracts with an environment. Path data is data characterizingobservation-action-reward tuples for each of multiple time steps as anagent interacts with an environment. “Reward” in this sense may refer toan indication of whether the agent has accomplished a task (e.g.,navigating to a target location in the environment) or of the progressof the agent towards accomplishing a task. On-policy path data refers topath data where the agent interacts with the environment by selectingactions based on the current action selection policy of thereinforcement learning system. On the other hand, off-policy path datarefers to any path data, including path data where the agent interactswith the environment by selecting actions based on an action selectionpolicy that is different from the action selection policy of thereinforcement learning system. Since the reinforcement learning systemas described in this specification can be trained using both on-policyand off-policy path data, it can be trained more quickly (e.g., overfewer training iterations) and achieve better performance (e.g., byenabling the agent to perform tasks more effectively) than reinforcementlearning systems that can only be trained using on-policy path data(e.g., reinforcement learning systems trained using policy-basedmethods).

Moreover, since the reinforcement learning system as described in thisspecification can be trained using off-policy path data, it can betrained using expert trajectories. Expert trajectories are off-policypath data generated by the interaction of an expert agent with theenvironment, where an expert agent is an agent that interacts with theenvironment in accordance with an action selection policy that isunknown but that causes the expert agent to effectively perform tasks.Since the reinforcement learning system as described in thisspecification can be trained with expert trajectories, it can be trainedmore quickly and achieve better performance than conventionalreinforcement learning systems that can only be trained using on-policypath data.

Moreover, since it can be trained more quickly, the reinforcementlearning system as described in this specification consumes fewercomputational resources during training (e.g., memory and computingpower) than conventional reinforcement learning systems that can only betrained using on-policy path data.

The reinforcement learning system as described in this specification canbe trained using a path consistency learning loss function that enablesstable parameter value updates (i.e., parameter value updates that arelikely to cause the system select actions that enable the agent toeffectively perform tasks). In contrast, conventional reinforcementlearning systems (e.g., systems trained using Q-learning methods) mayexhibit unstable behavior during training, unless extensivehyper-parameter tuning is performed to find a combination ofhyper-parameters that results in stability. Therefore, by reducing thenecessary amount of hyper-parameter tuning, the reinforcement learningsystem as described in this specification consumes fewer computationalresources during training than conventional reinforcement learningsystems.

The details of one or more embodiments of the subject matter of thisspecification are set forth in the accompanying drawings and thedescription below. Other features, aspects, and advantages of thesubject matter will become apparent from the description, the drawings,and the claims.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 shows an example reinforcement learning system.

FIG. 2 is a flow diagram of an example process for training areinforcement learning system based on a path consistency learning lossfunction.

FIG. 3 is a flow diagram of an example process for determining parametervalue updates based on a path consistency learning loss function.

Like reference numbers and designations in the various drawings indicatelike elements.

DETAILED DESCRIPTION

FIG. 1 shows an example reinforcement learning system 100. Thereinforcement learning system 100 is an example of a system implementedas computer programs on one or more computers in one or more locationsin which the systems, components, and techniques described below areimplemented.

The reinforcement learning system 100 selects actions 114 to beperformed by an agent 116 interacting with an environment 118 at each ofmultiple time steps. At each time step, the system 100 receives datacharacterizing the current state of the environment 118, e.g., an imageof the environment, and selects an action 114 to be performed by theagent 116 in response to the received data. Data characterizing a stateof the environment 118 will be referred to in this specification as anobservation 120.

The reinforcement learning system 100 described herein is widelyapplicable and is not limited to one specific implementation. However,for illustrative purposes, a small number of example implementations aredescribed below.

In some other implementations, the environment 118 is a real-worldenvironment and the agent 116 is a mechanical agent interacting with thereal-world environment. For example, the agent may be a robotinteracting with the environment to accomplish a specific task. Asanother example, the agent may be an autonomous or semi-autonomousvehicle navigating through the environment. In these implementations,the actions may be control inputs to control the robot or the autonomousvehicle. In some of these implementations, the observations 120 may begenerated by or derived from sensors of the agent 116. For example, theobservations 120 may be captured by a camera of the agent 116. Asanother example, the observations 120 may be derived from data capturedfrom a laser sensor of the agent 116. As another example, theobservations may be hyperspectral images captured by a hyperspectralsensor of the agent 116.

In some implementations, the environment 118 may be a simulatedenvironment and the agent 116 may be implemented as one or more computerprograms interacting with the simulated environment. For example, thesimulated environment may be a video game and the agent may be asimulated user playing the video game. As another example, the simulatedenvironment may be a motion simulation environment, e.g., a drivingsimulation or a flight simulation, and the agent is a simulated vehiclenavigating through the motion simulation environment. In theseimplementations, the actions may be control inputs to control thesimulated user or simulated vehicle.

At each time step, the state of the environment 118 at the time step (ascharacterized by the observation 120) depends on the state of theenvironment 118 at the previous time step and the action 114 performedby the agent 116 at the previous time step.

At each time step, the system 100 receives a reward 122 based on thecurrent state of the environment 118 and the action 114 of the agent 116at the time step. In general, the reward 122 is a numerical value. Thereward 122 can be based on any event or aspect of the environment 118.For example, the reward 122 may indicate whether the agent 116 hasaccomplished a task (e.g., navigating to a target location in theenvironment 118) or the progress of the agent 116 towards accomplishinga task.

The system 100 uses a policy neural network 112 in selecting actions tobe performed by the agent 116 in response to observations 120 at eachtime step. In particular, the policy neural network 112 is configured toreceive an observation 120 as input and to process the input inaccordance with a set of parameters, referred to in this specificationas policy neural network parameters, to generate a policy output thatthe system 100 uses to determine an action 114 to be performed by theagent 116 at the time step. The policy output is a score distributionthat includes a respective score for each action in a predetermined setof actions. In some cases, the system 100 determines the action 114 tobe performed by the agent 116 at the time step to be the action with thehighest score. In some other cases, the system 100 processes the scores(e.g., using a soft-max function) to generate a probability distributionover the predetermined set of actions. In these cases, the system 100determines the action 114 to be performed by the agent 116 at the timestep by sampling an action from the predetermined set of actions inaccordance with the probability distribution.

In some cases, the system 100 determines the action 114 to be performedby the agent 116 using an exploration strategy. For example, the system100 may use an ϵ-greedy exploration strategy. In this example, E is anumber between 0 and 1. The system 100 determines the action 114 to beperformed by the agent 116 based on the policy output generated by thepolicy neural network 112 with probability (1−ϵ), and determines theaction 114 to be performed by the agent 116 randomly with probability E.

The system 100 includes a training system 102 that is configured totrain the system 100 (including the policy neural network 112) overmultiple training iterations using reinforcement learning techniques.The training system 102 trains the system 100 to cause the policy neuralnetwork 112 to generate policy outputs that result in the selection ofactions 114 to be performed by the agent 116 which maximize a long-termtime-discounted reward (such as an expected entropy regularizedlong-term time-discounted reward) received by the system 100, andthereby cause the agent 116 to effectively perform given tasks.

The long-term time-discounted reward is a measure of the expected futurevalues of the rewards 122 received by the system 100, where the rewards122 are weighted by a discount factor reflecting the rewards 122received sooner are more valuable to the system 100 than rewards 122received later. In examples in which the long-term time-discountedreward is an expected entropy regularized long-term time-discountedreward, it also includes an entropy regularization term, that encouragesexploration and helps prevent the early convergence of the policy neuralnetwork 112 to sub-optimal action selection policies. The expectedentropy regularized long-term time-discounted reward received by thesystem 100 from a state s of the environment 118 can be recursivelyexpressed as:

${{\mathcal{O}\left( {s,\pi} \right)} = {\sum\limits_{a}^{\;}{{\pi\left( a \middle| s \right)}\left\lbrack {{r\left( {s,a} \right)} - {\tau\;\log\;{\pi\left( a \middle| s \right)}} + {\gamma\;{\mathcal{O}\left( {s^{\prime},\pi} \right)}}} \right\rbrack}}},$where the sum is over the predetermined set of actions a that can beperformed by the agent 116, π(a|s) is the score corresponding to actiona from the score distribution generated by policy neural network 112 inresponse to processing the observation of the environment in the states, r(s, a) is the reward received by the system 100 is the agent 116performs action a while the environment 118 is in state s, τ is ahyper-parameter governing the degree of entropy regularization, γ is adiscount factor, and s′ is the state of the environment 118 at the nexttime step if the current state of the environment 118 is s and the agent116 performs action a. In some implementations, the parameter τ is setto zero (or near zero), in which case the training system 102 trains thesystem 100 to cause the policy neural network 112 to generate policyoutputs that result in the selection of actions 114 to be performed bythe agent 116 which maximize the expected long-term time-discountedreward received by the system 100 (without entropy regularization).

The training system 102 includes a path sampling engine 108 that isconfigured to, at each training iteration, obtain path data that definesone or more paths traversed by an agent through the environment 118 overa predetermined number of time steps. For each path, the path dataincludes an observation-action-reward tuple for each time step of thepath, including the observation 120 at the time step, the action 114performed at the time step, and the reward 122 received at the timestep.

In some implementations, the path sampling engine 108 obtains offlinepath data from a replay memory 110. Offline path data is path data thatcan include paths traversed by an agent performing actions selectedbased on policy outputs that are different from those that would begenerated in accordance with the current values of the policy neuralnetwork parameters. The replay memory 110 stores path data from pathspreviously traversed by the agent 116, from paths previously traversedby an expert agent, or both. An expert agent is an agent that interactswith the environment 118 in accordance with an action selection policythat is unknown but that causes the expert agent to successfullyaccumulate long-term time-discounted rewards by effectively performingtasks. For example, if the system 100 receives rewards in response tothe agent 116 making progress towards accomplishing a task, an expertagent may be an agent controlled by a person who is skilled at the taskto be performed by the agent 116.

In some implementations, the path sampling engine 108 obtains the pathdata by generating path data on-policy. Generating path data on-policyrefers to selecting actions 114 to be performed by the agent 116 at eachof the predetermined number of time steps based on policy outputsgenerated in accordance with the current values of the policy neuralnetwork parameters. In some cases, the path sampling engine generatesmultiple different on-policy paths.

In some implementations, the path sampling engine 108 obtains path databoth by sampling from the replay memory 110 and by generating path dataon-policy.

The training system 102 includes a state value neural network 106 thatis configured to receive an observation 120 as input and to process theinput in accordance with a set of parameters, referred to in thisspecification as state value neural network parameters, to generate asoutput an estimated state value. A state value refers to the expectedentropy regularized long-term time-discounted reward that would bereceived by the system 100 if, starting from the current state of theenvironment 118, the agent 116 performs actions that are selected basedon the policy outputs generated by the policy neural network 112 inaccordance with the current values of the policy neural networkparameters, i.e., and not based on a different action selection policy.

The training system 102 includes a training engine 104 that isconfigured to, at each training iteration, update the current values ofthe policy neural network parameters and the state value neural networkparameters. In particular, the training engine 104 uses gradient descentto adjust the policy neural network parameters and the state valueneural network parameters to minimize a path consistency learning lossfunction 124. In general, a path consistency learning loss function(which will be described in more detail with reference to FIG. 2 andFIG. 3) is a loss function that includes consistency error terms thatmeasure a consistency between policy outputs (i.e., as generated by thepolicy neural network 112) and estimated state values (i.e., asgenerated by the state value neural network 106) for any path throughthe environment 118, and in particular, the paths of the path dataobtained by the path sampling engine 108.

By adjusting the policy neural network parameters and the state valueneural network parameters to minimize the path consistency learning lossfunction 124, the training engine 102 causes the policy neural network112 to generate policy outputs that, when used to select actions 114 tobe performed by the agent 116, maximize the expected entropy regularizedlong-term time-discounted reward received by the system 100 and therebycause the agent to effectively perform given tasks. An example processfor training the reinforcement learning system 100 is described withreference to FIG. 2.

In some implementations, the policy neural network 112 and the statevalue neural network 106 are implemented as separate neural networksthat do not share parameter values and are each separately trained bythe training engine 104. The policy neural network 112 and the statevalue neural network 106 may be implemented as any appropriate neuralnetwork model, such as a feed-forward neural network (e.g., amulti-layer perceptron or a convolutional neural network) or a recurrentneural network (e.g., a long short-term memory network).

In some other implementations, the system integrates the policy neuralnetwork 112 and the state value neural network 106 into a single model.In these implementations, rather than including a separate policy neuralnetwork 112 and state value neural network 106, the system 100 includesa neural network referred to in this specification as a Q neuralnetwork, and generates both the policy output (otherwise generated bythe policy neural network 112) and the estimated state values (otherwisegenerated by the state value network 106) based on the output of the Qneural network.

In some of these implementations, the Q neural network is configured toreceive as input an observation 120 and to process the input inaccordance with a set of parameters, referred to in this specificationas Q neural network parameters, to generate as output a differentestimated Q-value for each action of the predetermined set of actions.Given an observation, a Q-value for an action is a scalar value that isthe expected entropy regularized long-term time-discounted rewardreceived by the system 100 if the agent 116 first selects the givenaction (i.e., in response to the given observation) and then atsubsequent time steps selects actions in accordance with an optimalaction selection policy (i.e., an action selection policy that maximizesthe expected entropy regularized long-term time-discounted rewardreceived). In general, other Q neural network architectures arepossible. For example, the Q neural network may be configured to receiveas input both an observation and a representation of an action, and toprocess the inputs to generate as output an estimated Q-value for theparticular input action.

The system 100 generates the state value estimate for an observationfrom the output of the Q neural network as:

${V = {\tau\;\log{\sum_{a}\left( {\exp\frac{Q(a)}{\tau}} \right)}}},$where τ is the hyper-parameter governing the degree of entropyregularization (as described earlier), the sum is over the actions a inthe predetermined set of actions, and Q(a) is the estimated Q-value foraction a (i.e., generated by the Q neural network by processing theobservation).

The system 100 generates the policy output for an observation from theoutput of the Q neural network as:

${{\pi(a)} = {\exp\left\{ \frac{{Q(a)} - V}{\tau} \right\}}},$where π is the policy output (i.e., the score distribution over the setof predetermined actions), π(a) is the score for action a in the scoredistribution π, Q(a) is the estimated Q-value corresponding to action afor the observation, V is the state value estimate for the observation(determined from the output of the Q neural network as described above),and τ is the hyper-parameter governing the degree of entropyregularization (as described earlier).

When the system includes a Q neural network rather than a separatepolicy neural network 112 and state value neural network 106, thetraining engine 104 operates analogously, by updating the current valuesof the Q neural network parameters using gradient descent to adjust theQ neural network parameters to minimize the path consistency learningloss function 124.

The Q neural network may be implemented as any appropriate neuralnetwork model. For example, the Q neural network may be implemented as afeed-forward neural network (e.g., a multi-layer perceptron or aconvolutional neural network) or a recurrent neural network (e.g., along short-term memory network).

FIG. 2 is a flow diagram of an example process 200 for training areinforcement learning system based on a path consistency learning lossfunction. For convenience, the process 200 will be described as beingperformed by a system of one or more computers located in one or morelocations. For example, a training system, e.g., the training system 102of FIG. 1, appropriately programmed in accordance with thisspecification, can perform the process 200.

The system obtains path data (202). The path data defines one or moredifferent paths traversed by an agent through the environment over apredetermined number of time steps. For each path, the path dataincludes an observation-action-reward tuple for each time step,including the observation at the time step, the action performed at thetime step, and the reward received at the time step.

In some implementations, the system obtains offline path data from areplay memory. Offline path data is path data that can include pathstraversed by an agent performing actions selected based on policyoutputs that are different from those generated in accordance with thecurrent values of the policy neural network parameters. The replaymemory stores path data from paths previously traversed by the agentand/or from paths previously traversed by an expert agent. An expertagent is an agent that interacts with the environment in accordance withan action selection policy that is unknown but that causes the expertagent to successfully accumulate long-term time-discounted rewards. Forexample, if the system receives rewards in response to the agent makingprogress towards accomplishing a task, an expert agent may be an agentcontrolled by a person who is skilled at the task to be performed by theagent.

In some implementations, the system randomly samples path data from thereplay memory. In some implementations, the system determines aprobability distribution over the paths in the replay memory and samplespath data from the replay memory in accordance with the probabilitydistribution. In these implementations, the system may determine theprobability distribution by assigning a different probability to eachpath in the replay memory based on the reward data associated with thepath. For example, the system may assign a higher probability to pathswhere a measure of the total reward associated with the path (e.g., thesum of the rewards at each time step) is greater.

In some implementations, the system obtains the path data by generatingpath data on-policy. Generating path data on-policy refers to selectingactions to be performed by the agent at each of the predetermined numberof time steps based on policy outputs generated in accordance with thecurrent values of the policy neural network parameters. In some cases,the path sampling engine generates multiple different on-policy paths.

In some implementations, the system obtains path data both by samplingoffline path data from the replay memory and by generating path dataon-policy.

The system determines parameter value updates for (i) the policy neuralnetwork parameters and the state value neural network parameters, or(ii) the Q neural network parameters, depending on whether the policyneural network and the state value neural network are implemented asseparate models or integrated into a single model based on a Q neuralnetwork (as described earlier) (204).

The system is configured to determine parameter value updates for (i)the policy and state value neural network parameters, or (ii) the Qneural network parameters, to minimize a path consistency learning lossfunction. The path consistency learning loss function characterizes aconsistency between policy outputs and estimated state values for anypath through the environment, and in particular, the paths of theobtained path data. For example, the path consistency learning lossfunction may have the form:

${\mathcal{L} = {\sum\limits_{P \in E}^{\;}{\frac{1}{2}{C(P)}^{2}}}},$where P is a path from the set of obtained path data E, and C(P) is aconsistency error for path P. In general, the consistency error C(P) fora path P depends on the output of the policy neural network and thestate value neural network for observations in the path.

The system determines the parameter value updates by computing thegradient of the path consistency learning loss function with respect tothe parameters of (i) the policy and state value neural networks, or(ii) the Q neural network (depending on the implementation). An exampleprocess for determining parameter value updates based on a pathconsistency learning loss function is described with reference to FIG.3.

The system updates (i) the policy and state value neural networkparameters, or (ii) the Q neural network parameters (depending on theimplementation), using the determined parameter value updates (206).Specifically, the system updates the parameters of a network (e.g., apolicy, state value, or Q neural network) based on a determinedparameter value update by:θ←θ+η·Δθ,where θ are the parameters of the network, ← indicates the assignmentoperation, η is a learning rate (that controls the rate at which thevalues of the parameters are updated), and Δθ is the parameter valueupdate.

By adjusting the network parameters to minimize the path consistencylearning loss, the system causes policy outputs to be generated that,when used to select actions to be performed by the agent, maximize theexpected entropy regularized long-term time-discounted reward receivedby the system.

Optionally, the system updates the replay buffer by inserting one ormore paths from on-policy path data generated at the training iteration(e.g., as described in 202) (208). In some implementations, the systeminserts each generated on-policy path into the replay buffer. In someimplementations, the system inserts a generated on-policy path into thereplay buffer with a fixed probability, for example, 10%, or any otherappropriate probability. If the replay buffer is full, then the systemcan insert a given path into the replay buffer by selecting (e.g.,randomly) a path that is currently in the replay buffer and overwriting(i.e., replacing) it with the given path.

FIG. 3 is a flow diagram of an example process 300 for determiningparameter value updates based on a path consistency learning lossfunction. For convenience, the process 300 will be described as beingperformed by a system of one or more computers located in one or morelocations. For example, a training system, e.g., the training system 102of FIG. 1, appropriately programmed in accordance with thisspecification, can perform the process 300.

For each path of the obtained path data, the system determines acombined reward for the path based on the rewards received at one ormore time steps in the path (302). For example, the system may determinethe combined reward for a path to be the discounted sum of the rewardsreceived at each time step in the path except for the last time step,i.e.:

${\sum\limits_{j = 0}^{d - 1}{\gamma^{j}r_{j}}},$where the sum is over the time steps of the path except the last timestep, γ is the discount factor, and r_(j) is the reward received fortime step j in the path.

For each path of the obtained path data, the system determines estimatedstate values for the first and last observation of the path (304).

In some implementations, the system provides the first and lastobservation of each path as input to a state value neural network thatprocesses the input to generate as output respective estimated statevalues for the first and last observation of the path. In some otherimplementations, the system provides the first and last observation ofeach path as input to a Q neural network that processes the input togenerate as output respective estimated Q-values for each action in thepredetermined set of actions, for the first and last observation in thepath. In these implementations, the system determines the estimatedstate values for the first and last observation based on the respectiveQ-values, as described earlier.

For each path of the obtained path data, the system determines a pathlikelihood for the path (306).

For each path, the system determines a respective score distributionover the predetermined set of actions for each observation in the pathother than the last observation. In some implementations, the systemprovides each observation in the path other than the last observation asinputs to a policy neural network that processes the inputs to generateas output respective score distributions over the predetermined set ofactions for each observation in the path other than the lastobservation. In some other implementations, the system provides eachobservation in the path other than the last observation as input to a Qneural network that processes the inputs to generate as outputrespective estimated Q-values for each action in the predetermined setof actions, for each observation in the path other than the lastobservation. In these implementations, the system determines therespective score distributions for each observation in the path otherthan the last observation based on the respective Q-values, as describedearlier.

For each path, the system determines, for each observation in the pathother than the last observation, a selected action score. To determinethe selected action score for an observation in a path, the systemidentifies the score distribution over the set of predetermined actionsthat the system determined for the observation. The system identifiesthe selected action score as the score from the score distribution forthe action corresponding to the observation in the path according to thepath data.

For each path, the system determines the path likelihood for the pathbased on the selected action scores. For example, the system maydetermine the path likelihood for a path by:

${\sum\limits_{j = 0}^{d - 1}{\gamma^{j}\log\;{\pi\left( a_{j} \middle| P_{j} \right)}}},$where the sum is over each observation in the path P except for the lastobservation, γ is a discount factor, and π(a_(j)|P_(j)) is the selectedaction score for the observation P_(j) corresponding to the j-th timestep in the path.

For each path of the obtained path data, the system determines aconsistency error for the path (308). Specifically, the systemdetermines the consistency error for a path by combining at least (i)the combined reward for the path, (ii) the estimated state values forthe first and last observations in the path, and (iii) the pathlikelihood for the path. For example, the system may determine theconsistency error for a path P to be:

${{C(P)} = {{- {V\left( P_{0} \right)}} + {\gamma^{d}{V\left( P_{d} \right)}} + {\sum\limits_{j = 0}^{d - 1}{\gamma^{j}r_{j}}} - {\tau{\sum\limits_{j = 0}^{d - 1}{\gamma^{j}\log\;{\pi\left( a_{j} \middle| P_{j} \right)}}}}}},$where V(P₀) is the estimated state value estimate for the firstobservation in the path, V(P_(d)) is the estimated state value for thelast observation in the path, γ is a discount factor, (d+1) ispredetermined number of time steps in the path, r_(j) is the rewardreceived at the j-th time step in the path, τ is a hyper-parametergoverning the degree of entropy regularization (as described earlier),and π(a_(j)|P_(j)) is the selected action score for the j-th observationin the path.

The system determines parameter value updates for (i) the policy neuralnetwork parameters and the state value neural network parameters, or(ii) the Q neural network parameters, depending on whether the policyneural network and the state value neural network are implemented asseparate models or integrated into a single model based on a Q neuralnetwork (as described earlier) (310).

In general, the system determines the parameter value updates bycomputing the gradient of the path consistency learning loss functionwith respect to the parameters of (i) the policy and state value neuralnetworks, or (ii) the Q neural network (again, depending on theimplementation). As described earlier, the path consistency learningloss function is based on the consistency errors for each of the pathsof the obtained path data.

In implementations where the consistency error has the specific formdescribed above, then the parameter value updates for a policy neuralnetwork may have the form:

${{\Delta\theta} = {\sum\limits_{P \in E}^{\;}{\Delta\theta}_{P}}},{{\Delta\theta}_{P} = {{C(P)} \cdot {\nabla_{\theta}{\sum\limits_{j = 0}^{d - 1}{\gamma^{j}\log\;{\pi_{\theta}\left( a_{j} \middle| P_{j} \right)}}}}}},$where Δθ is the overall parameter value update, Δθ_(P) is the parametervalue update corresponding to path P from the set of obtained path dataE, C(P) is the consistency error for path P, ∇_(θ) is the gradient withrespect to the parameters of the policy neural network (e.g., computedby backpropagation), j is an index over the predetermined number of timesteps in each path, γ is a discount factor, and π_(θ)(α_(j)|P_(j)) isthe selected action score for the j-th observation in path P (where thedependence on the parameters θ of the policy neural network is madeexplicit).

In implementations where the consistency error has the specific formdescribed above, then the parameter value updates for a state valueneural network may have the form:

${{\Delta\phi} = {\sum\limits_{P \in E}^{\;}{\Delta\phi}_{P}}},{{\Delta\phi} = {{C(P)} \cdot {\nabla_{\phi}\left( {{V_{\phi}\left( P_{0} \right)} - {\gamma^{d}{V_{\phi}\left( P_{d} \right)}}} \right)}}},$where Δϕ is the overall parameter value update, Δϕ_(P) is the parametervalue update corresponding to path P from the set of obtained path dataE, C(P) is the consistency error for path P, ∇_(ϕ) is the gradient withrespect to the parameters of the state value neural network (e.g.,computed by backpropagation), ∇_(ϕ)(P₀) is the estimated state value(i.e., as generated by the state value neural network) for the firstobservation in the path, ∇_(ϕ)(P_(d)) is the estimated state value forthe last observation in the path (where the dependence on the parametersϕ of the state value neural network are made explicit), (d+1) is thenumber of predetermined steps in each path, and γ is a discount factor.

In implementations where the consistency error has the specific formdescribed above, then the parameter value updates for a Q neural networkmay have the form:

$\mspace{20mu}{{{\Delta\lambda} = {\sum\limits_{P \in E}^{\;}{\Delta\lambda}_{P}}},{{\Delta\lambda}_{P} = {{{C(P)} \cdot {\nabla_{\lambda}{\sum\limits_{j = 0}^{d - 1}{\gamma^{j}\log\;{\pi_{\lambda}\left( a_{j} \middle| P_{j} \right)}}}}} + {{C(P)} \cdot {\nabla_{\lambda}\left( {{V_{\lambda}\left( P_{0} \right)} - {\gamma^{d}{V_{\lambda}\left( P_{d} \right)}}} \right)}}}},}$where Δλ is the overall parameter value update, Δλ_(P) is the parametervalue update corresponding to path P from the set of obtained path dataE, C(P) is the consistency error for path P, A is the gradient withrespect to the parameters of the Q neural network (e.g., computed bybackpropagation), j is an index over the predetermined number of timesteps in each path, γ is a discount factor, π_(λ)(a_(j)|P_(j)) is theselected action score for the j-th observation in path P (where thedependence on the parameters A of the Q neural network is madeexplicit), V_(λ)(P₀) is the estimated state value for the firstobservation in the path, and V_(λ)(P_(d)) is the estimated state valuefor the last observation in the path (where the dependence on theparameters λ of the Q neural network is made explicit). In someimplementations, different learning rates (as described with referenceto 206) are applied to the first and second terms of the Q neuralnetwork parameter value update described above.

This specification uses the term “configured” in connection with systemsand computer program components. For a system of one or more computersto be configured to perform particular operations or actions means thatthe system has installed on it software, firmware, hardware, or acombination of them that in operation cause the system to perform theoperations or actions. For one or more computer programs to beconfigured to perform particular operations or actions means that theone or more programs include instructions that, when executed by dataprocessing apparatus, cause the apparatus to perform the operations oractions.

Embodiments of the subject matter and the functional operationsdescribed in this specification can be implemented in digital electroniccircuitry, in tangibly-embodied computer software or firmware, incomputer hardware, including the structures disclosed in thisspecification and their structural equivalents, or in combinations ofone or more of them. Embodiments of the subject matter described in thisspecification can be implemented as one or more computer programs, i.e.,one or more modules of computer program instructions encoded on atangible non-transitory storage medium for execution by, or to controlthe operation of, data processing apparatus. The computer storage mediumcan be a machine-readable storage device, a machine-readable storagesubstrate, a random or serial access memory device, or a combination ofone or more of them. Alternatively or in addition, the programinstructions can be encoded on an artificially-generated propagatedsignal, e.g., a machine-generated electrical, optical, orelectromagnetic signal, that is generated to encode information fortransmission to suitable receiver apparatus for execution by a dataprocessing apparatus.

The term “data processing apparatus” refers to data processing hardwareand encompasses all kinds of apparatus, devices, and machines forprocessing data, including by way of example a programmable processor, acomputer, or multiple processors or computers. The apparatus can alsobe, or further include, special purpose logic circuitry, e.g., an FPGA(field programmable gate array) or an ASIC (application-specificintegrated circuit). The apparatus can optionally include, in additionto hardware, code that creates an execution environment for computerprograms, e.g., code that constitutes processor firmware, a protocolstack, a database management system, an operating system, or acombination of one or more of them.

A computer program, which may also be referred to or described as aprogram, software, a software application, an app, a module, a softwaremodule, a script, or code, can be written in any form of programminglanguage, including compiled or interpreted languages, or declarative orprocedural languages; and it can be deployed in any form, including as astand-alone program or as a module, component, subroutine, or other unitsuitable for use in a computing environment. A program may, but neednot, correspond to a file in a file system. A program can be stored in aportion of a file that holds other programs or data, e.g., one or morescripts stored in a markup language document, in a single file dedicatedto the program in question, or in multiple coordinated files, e.g.,files that store one or more modules, sub-programs, or portions of code.A computer program can be deployed to be executed on one computer or onmultiple computers that are located at one site or distributed acrossmultiple sites and interconnected by a data communication network.

In this specification the term “engine” is used broadly to refer to asoftware-based system, subsystem, or process that is programmed toperform one or more specific functions. Generally, an engine will beimplemented as one or more software modules or components, installed onone or more computers in one or more locations. In some cases, one ormore computers will be dedicated to a particular engine; in other cases,multiple engines can be installed and running on the same computer orcomputers.

The processes and logic flows described in this specification can beperformed by one or more programmable computers executing one or morecomputer programs to perform functions by operating on input data andgenerating output. The processes and logic flows can also be performedby special purpose logic circuitry, e.g., an FPGA or an ASIC, or by acombination of special purpose logic circuitry and one or moreprogrammed computers.

Computers suitable for the execution of a computer program can be basedon general or special purpose microprocessors or both, or any other kindof central processing unit. Generally, a central processing unit willreceive instructions and data from a read-only memory or a random accessmemory or both. The essential elements of a computer are a centralprocessing unit for performing or executing instructions and one or morememory devices for storing instructions and data. The central processingunit and the memory can be supplemented by, or incorporated in, specialpurpose logic circuitry. Generally, a computer will also include, or beoperatively coupled to receive data from or transfer data to, or both,one or more mass storage devices for storing data, e.g., magnetic,magneto-optical disks, or optical disks. However, a computer need nothave such devices. Moreover, a computer can be embedded in anotherdevice, e.g., a mobile telephone, a personal digital assistant (PDA), amobile audio or video player, a game console, a Global PositioningSystem (GPS) receiver, or a portable storage device, e.g., a universalserial bus (USB) flash drive, to name just a few.

Computer-readable media suitable for storing computer programinstructions and data include all forms of non-volatile memory, mediaand memory devices, including by way of example semiconductor memorydevices, e.g., EPROM, EEPROM, and flash memory devices; magnetic disks,e.g., internal hard disks or removable disks; magneto-optical disks; andCD-ROM and DVD-ROM disks.

To provide for interaction with a user, embodiments of the subjectmatter described in this specification can be implemented on a computerhaving a display device, e.g., a CRT (cathode ray tube) or LCD (liquidcrystal display) monitor, for displaying information to the user and akeyboard and a pointing device, e.g., a mouse or a trackball, by whichthe user can provide input to the computer. Other kinds of devices canbe used to provide for interaction with a user as well; for example,feedback provided to the user can be any form of sensory feedback, e.g.,visual feedback, auditory feedback, or tactile feedback; and input fromthe user can be received in any form, including acoustic, speech, ortactile input. In addition, a computer can interact with a user bysending documents to and receiving documents from a device that is usedby the user; for example, by sending web pages to a web browser on auser's device in response to requests received from the web browser.Also, a computer can interact with a user by sending text messages orother forms of message to a personal device, e.g., a smartphone that isrunning a messaging application, and receiving responsive messages fromthe user in return.

Data processing apparatus for implementing machine learning models canalso include, for example, special-purpose hardware accelerator unitsfor processing common and compute-intensive parts of machine learningtraining or production, i.e., inference, workloads.

Machine learning models can be implemented and deployed using a machinelearning framework, e.g., a TensorFlow framework, a Microsoft CognitiveToolkit framework, an Apache Singa framework, or an Apache MXNetframework.

Embodiments of the subject matter described in this specification can beimplemented in a computing system that includes a back-end component,e.g., as a data server, or that includes a middleware component, e.g.,an application server, or that includes a front-end component, e.g., aclient computer having a graphical user interface, a web browser, or anapp through which a user can interact with an implementation of thesubject matter described in this specification, or any combination ofone or more such back-end, middleware, or front-end components. Thecomponents of the system can be interconnected by any form or medium ofdigital data communication, e.g., a communication network. Examples ofcommunication networks include a local area network (LAN) and a widearea network (WAN), e.g., the Internet.

The computing system can include clients and servers. A client andserver are generally remote from each other and typically interactthrough a communication network. The relationship of client and serverarises by virtue of computer programs running on the respectivecomputers and having a client-server relationship to each other. In someembodiments, a server transmits data, e.g., an HTML page, to a userdevice, e.g., for purposes of displaying data to and receiving userinput from a user interacting with the device, which acts as a client.Data generated at the user device, e.g., a result of the userinteraction, can be received at the server from the device.

While this specification contains many specific implementation details,these should not be construed as limitations on the scope of anyinvention or on the scope of what may be claimed, but rather asdescriptions of features that may be specific to particular embodimentsof particular inventions. Certain features that are described in thisspecification in the context of separate embodiments can also beimplemented in combination in a single embodiment. Conversely, variousfeatures that are described in the context of a single embodiment canalso be implemented in multiple embodiments separately or in anysuitable subcombination. Moreover, although features may be describedabove as acting in certain combinations and even initially be claimed assuch, one or more features from a claimed combination can in some casesbe excised from the combination, and the claimed combination may bedirected to a subcombination or variation of a subcombination.

Similarly, while operations are depicted in the drawings and recited inthe claims in a particular order, this should not be understood asrequiring that such operations be performed in the particular ordershown or in sequential order, or that all illustrated operations beperformed, to achieve desirable results. In certain circumstances,multitasking and parallel processing may be advantageous. Moreover, theseparation of various system modules and components in the embodimentsdescribed above should not be understood as requiring such separation inall embodiments, and it should be understood that the described programcomponents and systems can generally be integrated together in a singlesoftware product or packaged into multiple software products.

Particular embodiments of the subject matter have been described. Otherembodiments are within the scope of the following claims. For example,the actions recited in the claims can be performed in a different orderand still achieve desirable results. As one example, the processesdepicted in the accompanying figures do not necessarily require theparticular order shown, or sequential order, to achieve desirableresults. In some cases, multitasking and parallel processing may beadvantageous.

What is claimed is:
 1. A method of training a policy neural network usedto select actions to be performed by a reinforcement learning agentinteracting with an environment by performing actions from apre-determined set of actions, the policy neural network having aplurality of policy network parameters and being configured to processan input observation characterizing a current state of the environmentin accordance with the policy network parameters to generate a scoredistribution that includes a respective score for each action in thepre-determined set of actions, and the method comprising: obtaining pathdata defining a path through the environment traversed by the agent,wherein the path is a sequence from a first observation to a lastobservation, and wherein the path data includes a plurality ofobservation-action-reward tuples, wherein the observation in each tupleis an observation characterizing a state of the environment, the actionin each tuple is an action performed by the agent in response to theobservation in the tuple, and the reward in each tuple is a numericvalue received as a result of the agent performing the action in thetuple; determining a combined reward from the rewards in the tuples inthe path data; determining a first soft-max state value for the firstobservation in the path in accordance with current values of state valuenetwork parameters of a state value neural network; determining a lastsoft-max state value for the last observation in the path in accordancewith the current values of the state value network parameters of thestate value neural network; determining a path likelihood for the path,in accordance with current values of the policy parameters, based onlyon observations that are included in the path, comprising: for eachobservation in a set of multiple observations comprising onlyobservations that are included in the path: processing the observationusing the policy neural network to determine a respective scoredistribution in accordance with the current values of the policy networkparameters; and determining a selected action score, wherein theselected action score is the action score for the action in the sametuple as the observation in the score distribution for the observation;and determining the path likelihood for the path using the selectedaction scores corresponding to the observations in the set of multipleobservations comprising only observations included in the path;determining a consistency error for the path from the combined reward,the first and last soft-max state values, and the path likelihood;determining a gradient of the path likelihood with respect to the policyparameters; determining a value update for the current values of thepolicy parameters from the consistency error and the gradient; and usingthe value update to adjust the current values of the policy parameters.2. The method of claim 1, wherein determining the combined rewardcomprises determining a discounted sum of the rewards in the tuples inthe path.
 3. The method of claim 1, wherein the state value neuralnetwork is based on a Q neural network, and determining the firstsoft-max state value comprises: processing the first observation usingthe Q neural network in accordance with current values of Q networkparameters to generate a respective Q value for each action in thepredetermined set of actions; and determining the first soft-max statevalue V for the first observation that satisfies:${V = {\tau\;\log{\sum_{a}\left( {\exp\frac{Q(a)}{\tau}} \right)}}},$where the sum is over the actions a in the predetermined set of actions,Q(a) is a Q value output of the Q neural network for the action, and τis a constant value.
 4. The method of claim 1, wherein the state valueneural network is based on a Q neural network, and determining the lastsoft-max state value comprises: processing the last observation usingthe Q neural network in accordance with current values of Q networkparameters to generate a respective Q value for each action in thepredetermined set of actions; and determining the last soft-max statevalue V for the last observation that satisfies:${V = {\tau\;\log{\sum_{a}\left( {\exp\frac{Q(a)}{\tau}} \right)}}},$where the sum is over the actions a in the predetermined set of actions,Q(a) is a Q value output of the Q neural network for the action, and τis a constant value.
 5. The method of claim 1, wherein the policy neuralnetwork is based on a Q neural network, and processing an observationusing the policy neural network to generate a score distribution thatincludes a respective score for each action in the pre-determined set ofactions comprises: processing the observation using the Q neural networkin accordance with current values of Q network parameters to generate arespective Q value for each action in the predetermined set of actions;determining a soft-max state value for the observation that satisfies:${V = {\tau\;\log{\sum_{a}\left( {\exp\frac{Q(a)}{\tau}} \right)}}},$where the sum is over the actions a in the predetermined set of actions,Q(a) is a Q value output of the Q neural network for the action, and τis a constant value; and determining a score distribution π for theobservation that satisfies:${{\pi(a)} = {\exp\left\{ \frac{{Q(a)} - V}{\tau} \right\}}},$ where ais an action in the predetermined set of actions, π(a) is a score forthe action in the score distribution, Q(a) is a Q value output of the Qneural network for the action a, V is the soft-max state value for theobservation, and τ is a constant value.
 6. The method of claim 1,wherein determining the path likelihood for the path using the selectedaction scores corresponding to the observations in the set of multipleobservations comprising only observations included in the pathcomprises: determining a discounted sum of logarithms of the selectedaction scores, wherein the set of multiple observations comprises everyobservation included in the path other than the last observation in thepath.
 7. The method of claim 1, wherein the value update for the currentvalues of the policy parameters is a product of the consistency errorand the gradient.
 8. The method of claim 1, further comprising:determining a gradient of the first soft-max state value with respect tothe Q network parameters; determining a gradient of the last soft-maxstate value with respect to the Q network parameters; and determining avalue update for the current values of the Q network parameters from theconsistency error, the gradient of the first soft-max state value, andthe gradient of the last soft-max state value for the last observation;and using the value update to adjust the current values of the Q networkparameters.
 9. The method of claim 1, further comprising: generating thepath on-policy by selecting the actions to be performed in response tothe observations in the path using the policy neural network and inaccordance with the current values of the policy network parameters. 10.The method of claim 1, further comprising: sampling the path datadefining the path from a replay memory storing data generated as aresult of interactions of the agent with the environment.
 11. The methodof claim 1, further comprising: providing the trained policy neuralnetwork for use in selecting actions to be performed by thereinforcement learning agent interacting with the environment.
 12. Asystem comprising one or more computers and one or more storage devicesstoring instructions that when executed by the one or more computerscause the one or more computers to perform operations for training apolicy neural network used to select actions to be performed by areinforcement learning agent interacting with an environment byperforming actions from a pre-determined set of actions, the policyneural network having a plurality of policy network parameters and beingconfigured to process an input observation characterizing a currentstate of the environment in accordance with the policy networkparameters to generate a score distribution that includes a respectivescore for each action in the pre-determined set of actions, theoperations comprising: obtaining path data defining a path through theenvironment traversed by the agent, wherein the path is a sequence froma first observation to a last observation, and wherein the path dataincludes a plurality of observation-action-reward tuples, wherein theobservation in each tuple is an observation characterizing a state ofthe environment, the action in each tuple is an action performed by theagent in response to the observation in the tuple, and the reward ineach tuple is a numeric value received as a result of the agentperforming the action in the tuple; determining a combined reward fromthe rewards in the tuples in the path data; determining a first soft-maxstate value for the first observation in the path in accordance withcurrent values of state value network parameters of a state value neuralnetwork; determining a last soft-max state value for the lastobservation in the path in accordance with the current values of thestate value network parameters of the state value neural network;determining a path likelihood for the path in accordance with currentvalues of the policy parameters, based only on observations that areincluded in the path, comprising: for each observation in a set ofmultiple observations comprising only observations that are included inthe path: processing the observation using the policy neural network todetermine a respective score distribution in accordance with the currentvalues of the policy network parameters; and determining a selectedaction score, wherein the selected action score is the action score forthe action in the same tuple as the observation in the scoredistribution for the observation; and determining the path likelihoodfor the path using the selected action scores corresponding to theobservations in the set of multiple observations comprising onlyobservations included in the path; determining a consistency error forthe path from the combined reward, the first and last soft-max statevalues, and the path likelihood; determining a gradient of the pathlikelihood with respect to the policy parameters; determining a valueupdate for the current values of the policy parameters from theconsistency error and the gradient; and using the value update to adjustthe current values of the policy parameters.
 13. The system of claim 12,wherein determining the combined reward comprises determining adiscounted sum of the rewards in the tuples in the path.
 14. The systemof claim 12, wherein the state value neural network is based on a Qneural network, and determining the first soft-max state valuecomprises: processing the first observation using the Q neural networkin accordance with current values of Q network parameters to generate arespective Q value for each action in the predetermined set of actions;and determining the first soft-max state value V for the firstobservation that satisfies:${V = {\tau\;\log{\sum_{a}\left( {\exp\frac{Q(a)}{\tau}} \right)}}},$where the sum is over the actions a in the predetermined set of actions,Q(a) is a Q value output of the Q neural network for the action, and τis a constant value.
 15. The system of claim 12, wherein the state valueneural network is based on a Q neural network, and determining the lastsoft-max state value comprises: processing the last observation usingthe Q neural network in accordance with current values of Q networkparameters to generate a respective Q value for each action in thepredetermined set of actions; and determining the last soft-max statevalue V for the last observation that satisfies:${V = {\tau\;\log{\sum_{a}\left( {\exp\frac{Q(a)}{\tau}} \right)}}},$where the sum is over the actions a in the predetermined set of actions,Q(a) is a Q value output of the Q neural network for the action, and τis a constant value.
 16. The system of claim 12, wherein the policyneural network is based on a Q neural network, and processing anobservation using the policy neural network to generate a scoredistribution that includes a respective score for each action in thepre-determined set of actions comprises: processing the observationusing the Q neural network in accordance with current values of Qnetwork parameters to generate a respective Q value for each action inthe predetermined set of actions; determining a soft-max state value forthe observation that satisfies:${V = {\tau\;\log{\sum_{a}\left( {\exp\frac{Q(a)}{\tau}} \right)}}},$where the sum is over the actions a in the predetermined set of actions,Q(a) is a Q value output of the Q neural network for the action, and τis a constant value; and determining a score distribution π for theobservation that satisfies:${{\pi(a)} = {\exp\left\{ \frac{{Q(a)} - V}{\tau} \right\}}},$ where ais an action in the predetermined set of actions, π(a) is a score forthe action in the score distribution, Q(a) is a Q value output of the Qneural network for the action a, V is the soft-max state value for theobservation, and τ is a constant value.
 17. One or more non-transitorycomputer storage media storing instructions that when executed by one ormore computers cause the one or more computers to perform operations fortraining a policy neural network used to select actions to be performedby a reinforcement learning agent interacting with an environment byperforming actions from a pre-determined set of actions, the policyneural network having a plurality of policy network parameters and beingconfigured to process an input observation characterizing a currentstate of the environment in accordance with the policy networkparameters to generate a score distribution that includes a respectivescore for each action in the pre-determined set of actions, theoperations comprising: obtaining path data defining a path through theenvironment traversed by the agent, wherein the path is a sequence froma first observation to a last observation, and wherein the path dataincludes a plurality of observation-action-reward tuples, wherein theobservation in each tuple is an observation characterizing a state ofthe environment, the action in each tuple is an action performed by theagent in response to the observation in the tuple, and the reward ineach tuple is a numeric value received as a result of the agentperforming the action in the tuple; determining a combined reward fromthe rewards in the tuples in the path data; determining a first soft-maxstate value for the first observation in the path in accordance withcurrent values of state value network parameters of a state value neuralnetwork; determining a last soft-max state value for the lastobservation in the path in accordance with the current values of thestate value network parameters of the state value neural network;determining a path likelihood for the path in accordance with currentvalues of the policy parameters, based only on observations that areincluded in the path, comprising: for each observation in a set ofmultiple observations comprising only observations that are included inthe path: processing the observation using the policy neural network todetermine a respective score distribution in accordance with the currentvalues of the policy network parameters; and determining a selectedaction score, wherein the selected action score is the action score forthe action in the same tuple as the observation in the scoredistribution for the observation; and determining the path likelihoodfor the path using the selected action scores corresponding to theobservations in the set of multiple observations comprising onlyobservations included in the path; determining a consistency error forthe path from the combined reward, the first and last soft-max statevalues, and the path likelihood; determining a gradient of the pathlikelihood with respect to the policy parameters; determining a valueupdate for the current values of the policy parameters from theconsistency error and the gradient; and using the value update to adjustthe current values of the policy parameters.
 18. The non-transitorycomputer storage media of claim 17, wherein determining the combinedreward comprises determining a discounted sum of the rewards in thetuples in the path.
 19. The non-transitory computer storage media ofclaim 17, wherein the state value neural network is based on a Q neuralnetwork, and determining the first soft-max state value comprises:processing the first observation using the Q neural network inaccordance with current values of Q network parameters to generate arespective Q value for each action in the predetermined set of actions;and determining the first soft-max state value V for the firstobservation that satisfies:${V = {\tau\;\log{\sum_{a}\left( {\exp\frac{Q(a)}{\tau}} \right)}}},$where the sum is over the actions a in the predetermined set of actions,Q(a) is a Q value output of the Q neural network for the action, and τis a constant value.
 20. The non-transitory computer storage media ofclaim 17, wherein the state value neural network is based on a Q neuralnetwork, and determining the last soft-max state value comprises:processing the last observation using the Q neural network in accordancewith current values of Q network parameters to generate a respective Qvalue for each action in the predetermined set of actions; anddetermining the last soft-max state value V for the last observationthat satisfies:${V = {\tau\;\log{\sum_{a}\left( {\exp\frac{Q(a)}{\tau}} \right)}}},$where the sum is over the actions a in the predetermined set of actions,Q(a) is a Q value output of the Q neural network for the action, and τis a constant value.